PAC-Bayes-Empirical-Bernstein Inequality
نویسندگان
چکیده
We present a PAC-Bayes-Empirical-Bernstein inequality. The inequality is based on a combination of the PAC-Bayesian bounding technique with an Empirical Bernstein bound. We show that when the empirical variance is significantly smaller than the empirical loss the PAC-Bayes-Empirical-Bernstein inequality is significantly tighter than the PAC-Bayes-kl inequality of Seeger (2002) and otherwise it is comparable. Our theoretical analysis is confirmed empirically on a synthetic example and several UCI datasets. The PAC-Bayes-Empirical-Bernstein inequality is an interesting example of an application of the PAC-Bayesian bounding technique to self-bounding functions.
منابع مشابه
PAC-Bayes-Bernstein Inequality for Martingales and its Application to Multiarmed Bandits
We combine PAC-Bayesian analysis with a Bernstein-type inequality for martingales to obtain a result that makes it possible to control the concentration of multiple (possibly uncountably many) simultaneously evolving and interdependent martingales. We apply this result to derive a regret bound for the multiarmed bandit problem. Our result forms a basis for integrative simultaneous analysis of e...
متن کاملPAC-Bayesian Analysis of the Exploration-Exploitation Trade-off
We develop a coherent framework for integrative simultaneous analysis of the explorationexploitation and model order selection tradeoffs. We improve over our preceding results on the same subject (Seldin et al., 2011) by combining PAC-Bayesian analysis with Bernstein-type inequality for martingales. Such a combination is also of independent interest for studies of multiple simultaneously evolvi...
متن کاملHands-On Learning Theory Fall 2016, Lecture 4
Recall that in Theorem 2.1, we analyzed empirical risk minimization with a finite hypothesis class F , i.e., |F| < +∞. Here, we will prove results for possibly infinite hypothesis classes. Although the PAC-Bayes framework is far more general, we will concentrate of the prediction problem as before, i.e., (∀f ∈ F) f : X → Y. Also, note that Theorem 2.1 could have been stated in a more general fa...
متن کاملHands-On Learning Theory Fall 2017, Lecture 4
Recall that in Theorem 2.1, we analyzed empirical risk minimization with a finite hypothesis class F , i.e., |F| < +∞. Here, we will prove results for possibly infinite hypothesis classes. Although the PAC-Bayes framework is far more general, we will concentrate of the prediction problem as before, i.e., (∀f ∈ F) f : X → Y. Also, note that Theorem 2.1 could have been stated in a more general fa...
متن کاملGaussian Processes Classification and its PAC-Bayes Generalization Error Bounds – CSE 291 Project Report
McAllester’s PAC-Bayes theorem (strengthened by [4]) characterizes the convergence of a stochastic classifier’s empirical error to its generalization error. Fixed one ”prior” distribution P (h) over hypothesis space H, the theorem can hold for all ”posterior” distribution Q(h) over H simultaneously, so in practice we can find a data-dependent posterior distribution overH as the distribution of ...
متن کامل